As is known in the art, Quality of Service (QoS) is a general heading that can include any of several different features in a network. One such feature is to provide a guarantee of a minimum allocated bandwidth through a network for a call or a set of calls. A call is defined as communication from one end node to another end node. A call may consist of one or more connections, where a connection is defined as communication from the ingress of one port within a switching device to the egress of a port, within the same switching device. A “flow” is defined as a one distinguishable communication flow from one end node to another end node. A call consists of one or more flows. For example, two end nodes may be communicating according to two different applications, e.g. a database transaction application and a data replication application. The two end nodes have one established call in each direction, and each call consists of two distinguishable flows. The terms “call,” “connection,” and “flow” typically imply connection-oriented service, where resources are dedicated for periods of time to the communication between the two communicating entities, but in the description of the present invention, the terms are used more broadly, without requiring connection-orientation. Another QoS feature is to provide a maximum bandwidth that may be used by a connection or a set of connections. Yet another is policing, whereby a switch, router, or other network appliance ensures that the service level agreement (SLA) for each connection is not violated by the sources, and the term “policing” usually refers to dropping frames in order to maintain SLAs.
Flow control mechanisms may also be used for SLA maintenance. QoS may also include guarantees that the latency for frames will not exceed a threshold. QoS may also include isochrony guarantees, meaning that frames can be expected to arrive at the destination at regular (to within some tolerance) intervals, at least if the frames are transmitted by the source at regular intervals. End users typically prefer absolute QoS guarantees to statistical guarantees. For example, an end user may prefer to be ensured that a connection will always be granted a certain minimum bandwidth or more, rather than being guaranteed that the connection will be granted at least the desired minimum bandwidth 95% of the time that the switch is operational.
Queuing and scheduling are important for enabling QoS. Some systems funnel all incoming frames into a single first-in first-out (FIFO) queue, but this can result in blocking if the frame at the head of the queue cannot be serviced. Output queuing is often considered the ideal, and is used as a reference for performance of other methods. With output queuing, each frame is immediately forwarded from the input port (ingress) to the appropriate output port (egress). Each output port must have sufficient bandwidth to handle frames being simultaneously forwarded from multiple port ingresses. The output port must also have sufficient buffering and/or adequate flow control to handle offered loads greater than unity. Switches designed to switch high-speed traffic typically cannot be designed using output queuing because of the memory bandwidth requirements at the egress ports.
A popular alternative is virtual output queuing. Each ingress has a set of virtual output queues (VOQs). As frames come into the ingress, they are inspected and fed into the appropriate VOQ. A VOQ might be assigned to each egress. Multiple VOQs could be assigned to each egress to further differentiate traffic flows. The switching fabric internal to the switch can be scheduled at regular intervals to define a set of nonconflicting ingress-egress pairs, enabling frames to move from ingress to egress. The scheduling can be used to ensure that the rate of frame transmission to the egress does not exceed the nominal capacity of the egress. Many scheduling/arbitration algorithms in use in commercial products, and described in the open literature, are unsuitable for ensuring absolute QoS guarantees.
If an ingress has multiple queues per egress, a queuing discipline (scheduling at the port ingress, or for a group of ports, rather than general scheduling of the entire internal switch fabric) may also be needed in order to select from which queue the ingress should send to the egress at a particular time. Many systems implemented in commercial products and described in the literature are unsuitable for ensuring absolute QoS guarantees.
Fibre Channel (FC) has been standardized by the American National Standards Institute (ANSI). FC is defined by Technical Committee T11, the committee within the InterNational Committee for Information Technology Standards (INCITS) responsible for device level interfaces. INCITS is accredited by, and operates under rules approved by the American National Standards Institute (ANSI). FC is a serial data transfer architecture that has been widely adopted in storage area networks (SANs). FC is well suited to storage environments because of the efficiency and reliability with which it transports blocks of data suited to storage environments.
Within the FC architecture, three topologies are defined: point-to-point, arbitrated loop (FC_AL), and fabric. Point-to-point is a direct, dedicated link between two end nodes (N_Ports). Because there are no shared resources, no sophisticated queuing or scheduling is required.
FC_AL is a shared topology, where multiple devices are attached to a loop and each must arbitrate for access at a given time. FC_AL defines arbitration mechanisms and allows some flexibility. Some additional QoS could be overlaid on FC_AL. FC_AL is commonly used within large storage arrays.
The fabric topology is general, and supports switches that can interconnect multiple devices, with multiple flows sending frames at the same time. Fabric also supports meshes of switches. Because of its generality, most of the examples in the descriptions of the preferred embodiments of the present invention assume fabric topology. However, those skilled in the art will be able to apply some of the methods to FC_AL as well.
FC offers multiple classes of service. These include:    Class 1—connection-oriented, dedicated path    Class 2—acknowledged connectionless    Class 3—unacknowledged connectionless    Class 4—connection-oriented, fractional bandwidth, requiring hardware modifications.
Fibre Channel Class 1 service dedicates an entire path to one call in one direction. While this ensures that the call receives all of the resources available, it can be inefficient not to share some of the bandwidth on a path with other calls. Thus, Class 1 is not often used.
Class 4 service is a more efficient alternative to Class 1, in that it enables dedicated fractional bandwidth service. Calls are guaranteed a fixed amount of bandwidth on each leg of the communication path from source to destination, and this bandwidth can be a fraction of, rather than the entire bandwidth of each segment. The drawback of Class 4 is that it requires changes to both the switch/router and host bus adapter (HBA) hardware. The industry has not adopted Class 4. Most FC users use Class 3 service (described below). Some use Class 2 service (described below). The unavailability of Class 4 HBAs and switches eliminated Class 4 from consideration for most customers. Although Class 4 would provide a solid infrastructure for QoS, it is not a practical starting point for a switch designer. Therefore, the description of the present invention focuses on the overlay of QoS onto Class 3 service. Those skilled in the art will be able to generalize the methods herein to other classes of service.
Class 3 service is in a sense the simplest: unacknowledged connectionless service. A transmitter transmits data basically whenever it wishes, as permitted by FC flow control, and without any feedback from the network or the receiver indicating the success of the transmission.
Class 2 service is similar to Class 3 in being connectionless, but Class 2 is an acknowledged service. That is, the recipient sends small messages back to the transmitter indicating the success of the receipt of data frames, so as to improve reliability.
While FC offers several different classes of service, there are users who would prefer more options for quality of service. In particular, there are customers who would use the fractional bandwidth capabilities of Class 4, e.g. if Class 4 hardware were commercially available or if the use of Class 4 service did not require a major hardware replacement. Users have expressed a need to be able to allocate a minimum amount of bandwidth to a particular call, or to be able to enforce a maximum bandwidth to be used by a call or by a set of calls in Fibre Channel based networks. Frame drops (loss of frames) is generally included under the heading of “QoS” but Fibre Channel flow control was designed to avoid frame drops due to buffer overflow and most Fibre Channel network users will not tolerate frame drops. Other QoS parameters can include latency and variance or isochrony, which are important to certain applications, including some replication and video distribution applications.
Overprovisioning, which is network design ensuring that the nominal fabric bandwidth exceeds the anticipated sustained load, is common practice in Fibre Channel SAN design. Designers often assume that overprovisioning will increase the probability that applications will receive the bandwidth required. However, typical deployments involve “many-to-one” scenarios, where multiple hosts or servers send or receive data to or from a single storage port. Having excess switch bandwidth may not offer any benefits because congestion at the storage ports (and attached switch ports) cause performance problems. A switch that can maintain SLAs under congestion may be more useful effective, and may be less costly than overprovisioning.
Fibre Channel uses a flow control mechanism whereby a device port sends “buffer-to-buffer credits” by means of R_RDY primitive sequences to the port at the other end of the link. If a device port has ingress buffer space adequate to hold M maximum-length FC frames, then that device will send up to M R_RDYs to the attached device. For each R_RDY received by the attached device, it is permitted to send a frame. The attached device counts outstanding R_RDYs, and if the number drops to zero, the attached device must stop sending frames to the port until more R_RDYs are sent by that port.
Ethernet is another standard used for networking. Ethernet is ubiquitous in local area network (LAN) environments, and there are efforts in standards bodies and in the commercial realm to grow the sphere of influence of Ethernet to metropolitan, access, and storage area networks. The widespread internet protocol (IP) is typically delivered using Ethernet. In the SAN arena, iSCSI is an emerging protocol that utilizes both IP and Ethernet. A switch that can pass Ethernet traffic can also pass iSCSI traffic, but generally cannot provide the protocol translation for stripping off the Ethernet, IP, and iSCSI headers to interface directly to a SCSI device. Ethernet does not provide QoS intrinsically. The methods taught herein enable QoS on an Ethernet network.
Although Ethernet does not specify a flow control that is exactly analogous to Fibre Channel buffer-to-buffer credit mechanism, the PAUSE frame mechanism bears significant similarities and can be used in a similar manner to avoid buffer overruns.